Papers with KV Cache
AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models require substantial storage space to perform tasks such as text generation and video generation. |
| Approach: | They propose to compress large language models using integer replacements for floating-point numbers, in a process known as Quantization. |
| Outcome: | The proposed model allows for quantization of up to 75% decoder layers with 1 bit while maintaining performance levels comparable to those of the models with floating parameters. |
NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time (2024.acl-long)
Copied to clipboard
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu
| Challenge: | Large Language Models (LLMs) with extended context windows are expensive and infeasible on fixed memory hardware due to the surprisingly large memory consumption of KV Cache. |
| Approach: | They propose a general framework for long-context KV cache eviction that achieves more optimal and efficient evict in a single operation during the encoding phase. |
| Outcome: | The proposed framework improves performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 5 with over 95% performance maintenance. |
Accurate KV Cache Quantization with Outlier Tokens Tracing (2025.acl-long)
Copied to clipboard
Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
| Challenge: | Large Language Models (LLMs) require substantial computational resources during deployment. |
| Approach: | They propose a method to identify outlier tokens and exclude them from quantization . they find that the method can deliver a 6.4 times reduction in memory usage and a 2.5 times increase in throughput . |
| Outcome: | The proposed method delivers a 6.4 times reduction in memory usage and a 2.5 times increase in throughput under 2-bit quantization. |
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for cache compression are heuristic and lack dynamic budget allocation . cnn's john mccartney and johnny mccain present a new approach for cache eviction and dynamic budgets . |
| Approach: | They propose a unified framework for cache compression that minimizes information loss in transformer residual streams. |
| Outcome: | The proposed method consistently maintains top performance across task types. |